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to  1-P,  where  P  is  the  transition  matrix  of  the  discrete  time  junq>  Markov 
chain . 


ABSTRACT 


The  main  aim  of  this  paper  is  to  examine  the  applicability  of  generalized 
inverses  to  a  wide  variety  of  problems  in  applied  probability  where  a  Markov 
chain  is  present  either  directly  or  indirectly  through  some  form  of  imbedding. 
By  characterizing  all  generalized  inverses  of  I-P,  where  P  is  the  transition 
matrix  of  a  finite  irreducible  discrete  time  Markov  chain,  we  are  able  to 
obtain  general  procedures  for  finding  stationary  distributions,  moments  of  the 
first  passage  time  distributions  and  asymptotic  forms  for  the  moments  of  the 
occupation  time  random  variables.  It  is  shown  that  all  known  explicit  methods 
for  examining  these  problems  can  be  expressed  in  this  generalized  inverse 
framework.  More  generally,  in  the  context  of  a  Markov  renewal  process  setting 
the  aforementioned  problems  are  also  examined  using  generalized  Inverses  of 
I-P.  As  a  special  case  Markov  chains  in  continuous  time  are  considered  and  we 
show  that  the  generalized  inverse  technique  can  be  applied  direct  to  the 
infinitesimal  generator  of  the  process.  Instead  of  to  I-P,  where  P  is  the 
transition  matrix  of  the  discrete  time  jump  Markov  chain. 
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1.  Introduction 


In  many  stochastic  modelling  situations  a  Markov  chain  in  discrete  time 
is  often  present,  either  directly  or  indirectly  through  some  form  of  imbedding. 
The  most  general  framework  that  has  been  devised  for  handling  such  situations 
is  the  concept  of  a  Markov  renewal  process  together  with  its  associated  semi- 
Markov  process.  Invariably  in  an  attempt  to  extract  information  concerning 
the  behavior  of  such  processes  (either  for  the  Markov  chain  in  discrete  time 
or  for  the  more  general  continuous  time  process)  we  are  interested  in  obtain¬ 
ing;  a)  stationary  distributions,  b)  expressions  for  the  moments  of  the  first 
passage  time  distributions  and  c)  asymptotic  forms  for  the  expected  nund>er  of 
visits  to  each  state  (occupation  time  random  variables  in  the  Markov  chain 
setting  or,  more  generally,  the  Markov  renewal  counting  random  variables) . 

In  section  2  we  discuss  these  three  general  problem  areas  and  show  that 
they  are  all  intimately  related  to  the  form  and  structure  of  P,  the  transition 
matrix  of  the  imbedded  discrete  time  Markov  chain. 

Quite  often,  once  some  preliminary  analysis  has  been  carried  out,  we  find 
that  the  properties  of  interest  can  be  obtained  by  solving  a  system  of  linear 
equations  Involving  P,  or  more  particularly  I-P  where  I  is  the  identity  matrix. 
One  condition  generalized  inverses  (commonly  called  g-inverses)  are  shown,  in 
section  3,  to  be  extremely  useful  in  handling  such  types  of  equations. 

Although  g-inverses  are  not  necessarily  unique  we  are  able  to  characterize  all 
g-lnverses  of  I-P  when  P  is  finite  and  irreducible. 

In  the  remaining  three  sections  of  the  paper  the  problems  a) ,  b)  and  c) 
alluded  to  earlier  are  investigated  for  Markov  chains  in  discrete  time,  Markov 
renewal  processes  (or  the  associated  semi-Markov  process)  and  Markov  chains  in 
continuous  time.  In  this  latter  process,  a  special  Markov  renewal  process,  its 


stochastic  behavior  is  governed  by  an  infinitesimal  generator  Q  and  we  show 
that  the  generalized  inverse  technique  can  be  applied  direct  to  Q  instead  of 
to  1-P  where  P  is  the  transition  matrix  of  the  imbedded  discrete  time  "jump" 
chain. 

No  numerical  or  computational  studies  have  been  carried  out  using  the 
methods  presented  but  there  are  some  compelling  reasons  why  the  procedures 
given  in  the  paper  should  prove  to  be  useful.  Firstly,  many  computer  sub¬ 
routines  for  producing  generalized  inverses  are  now  available  and  since  many 
of  the  techniques  developed  herein  require  no  special  form  for  the  generalized 
inverse  any  such  package  can  be  used  without  any  modification.  Secondly,  if 
some  additional  information  is  known  concerning  the  structure  of  the  generalized 
inverse  in  the  subroutine  the  solution  of  the  problem  at  hand  may  be  considerably 
simplified.  Furthermore,  any  known  explicit  methods  for  solving  these  problems, 
that  have  already  appeared  in  the  literature,  can  be  expressed  in  the  framework 
presented  in  this  paper.  It  is  the  universality  of  the  results  presented  herein 
that  should  prove  to  be  valuable. 

2.  Models  and  Problems  of  Interest 
2.1.  Markov  chains  in  discrete  time 

Let  {Xn},  (n^  0),  be  a  discrete  time,  finite,  homogeneous,  Markov  chaih 
with  state  space  S  taken  to  be  {l,2,”*,m}.  Let  P  *  [p±^  ]  be  the  transition 
matrix  of  the  chain  which,  for  the  rest  of  the  paper,  will  be  assumed  to  be 
irreducible.  He  make  no  assumption  whether  the  Markov  chain  is  aperiodic  or 
periodic  and  the  analysis  presented  covers  both  of  these  cases. 

It  is  well  known  (Feller  [6],  Kemeny  and  Snell  [  9])  that  for  such  chains 
a  stationary  (or  invariant)  probability  distribution  {i^}  (i  *  l,2,”*,m)  exists. 


is  unique,  and  satisfies  the  equations 


Jj 


m  m 

(2.1)  n .  ■  E  x.p.  .  and  1  tt.  ■  1. 

j  i*l  1  1=1  1 

If  *’  -  (ir^.TT^,*  *  *  .it^)  and  e'  -  (1,1,***,1),  equation  (2.1)  can  be 
expressed  as 

(2.2)  it’ (I  -  P)  ■  0'  subject  to  x'e  *  1. 

Thus  the  stationary  probability  vector  y'  can  be  determined  by  solving 
a  constrained  system  of  linear  equations  involving  I  -  P,  a  singular  matrix. 
We  examine  general  procedures  for  solving  such  a  system  in  section  3.2  and 
apply  these  procedures  to  the  problem  at  hand  in  section  4.1. 

Let  be  the  number  of  trials  for  a  first  passage  from  state  i  to 
state  j  (first  return  if  i  =■  j),  i.e., 

-  min{n:  X^  -  j|xQ  »  i>. 

Under  the  assumption  of  irreducibility  for  finite  chains  the  are  proper 
random  variables,  the  moments  m^  =  E  are  well  defined  and  finite  for 
all  i,j  e  S  (cf..  Theorem  13.4.1  of  Neuts  [14]).  Furthermore,  it  can  be 
shown  (Kemeny  and  Snell  [  9 ])  that  the  mean  first  passage  time  from  state 
i  to  state  j,  m^j  =  m^  satisfies  the  equation 

(2.3)  ,l]‘1+Jj,ilV(t’J,S)' 

If  we  use  the  notation  M  ■  [m^],  M<j  “  [6^  and  E  “  UJ  (i.e.,  all 

the  elements  of  E  are  unity),  equations  (2.3)  can  be  expressed  as 


3 


(2.4) 


(I  -  P)M  -  E  -  PMd. 

Once  again  we  have  a  system  of  linear  equations  involving  the  matrix 
I  -  P.  As  we  shall  see,  (section  3.2),  when  we  solve  such  equations  involving 
singular  matrices  there  is  some  arbitrariness  in  the  solution.  However,  the 
right  hand  side  of  equation  (2.4)  involves  M.  which  can  be  determined  in  terms 

a 

of  n'  and  this  turns  out  to  be  sufficient  to  enable  us  to  obtain  an  explicit 

solution  for  M.  In  fact  is  is  well  known  (Feller  [  6  ),  Kemeny  and  Snell  [ 9  ]) 

that  if  it'  *  (ir. •  *,ir  )  is  the  stationary  probability  vector  then 
~  x  l  m 

m  ^  »  1/itj  and  thus  if  n  *  ex' , 

(2.5)  Md  -  (nd)-1. 


General  solutions  to  equation  (2.4)  are  discussed  in  section  5.1. 

It  is  also  possible  to  derive  expressions  for  the  matrices  of  higher 
(k)  (k) 

moments  M  *  [m^  ].  Since  a  Markov  chain  is  a  special  Markov  renewal 
process  such  problems  can  be  considered  in  the  more  general  Markov  renewal 


setting. 

Let  be  the  number  of  k  (0  k  _<  n)  and  he  the  number  of  k 
(1  <.  k  <  n)  such  that  X^  *  J  given  Xq  ■  i.  Thus  M^  *  +  6^  and 

these  random  variables  record  the  number  of  visits  the  Markov  chain  makes  to 
state  j  by  trial  n  with  M^”^  recording  the  initial  occurrence  at  the  zeroth 


time  if  i  ■  j.  There  is  no  standard  terminology  for  distinguishing  between 

these  random  variables.  We  shall  call  the  occupancy  time  random 

variable  and  the  modified  occupancy  time  random  variable. 

(k) 

If  pjj  is  the  k  step  transition  probability  from  state  i  to  state  j 
<PiJ>  -  -  P^)  and  if  P(k)  -  [p<k)]  then  P(k)  -  Pk,  (k  >  1), 
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We  shall  show  In  section  6.1  that  we  can  utilize  generalized  inverses 
of  I  -  P  to  obtain  a  variety  of  expressions  for  [E  ]  and  [EN^^]. 

2.2.  Markov  renewal  processes  and  semi-Markov  processes 

Let  {(xn»^n)}  (n  >  0)  be  a  Markov  renewal  process  with  state  space 
S  =■  {1,2,  •  •  •  ,m}  and  semi-Markov  kernel  Q(t)  *  [Q^Ct)]  where 

V*>  ‘  mn*l  '  J>  Vl  -  T„  i  -  1]-  *"i  ‘  S' 

The  terminology  used  above  is  that  which  was  introduced  by  £inlar  14]. 
For  a  descriptive  definition  see  Hunter  [8].  One  way  of  visualizing  a  Markov 
renewal  process  is  to  interpret  Xr  as  the  state  the  stochastic  process  is  in 
following  the  nth  transition  and  Tn  is  the  time  that  the  nth  transition 
occurs,  (0  -  Tq  <  ^  _<  T2  <  •••)• 

One  pertinent  observation  is  that  the  transitions  are  governed  by  a 
discrete  time  Markov  chain  {Xn>  with  transition  matrix  P*  [p^j  1  *  [Q^j (+*)]. 
Given  that  the  (n+l)th  transition  takes  the  process  from  state  i  to  state  j, 
the  time  between  transitions  is  governed  by  a  distribution  function 


'  P'Vl  -  Tn  i  CIX„  '  ‘*  Vl  '  J1' 
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Thus,  provided  p±J  >  0,  Q±j<t)  -  p^  F^t).  (If  p^  =  0  then  (^(t)  -  0 

for  all  t  and  F^(t)  may  be  arbitrarily  defined,  say  Fij(t)  *  1  for  t  >  0.) 

Let  H. (t)  •  PfT  -  -  T  <  tlx  *  i]  be  the  distribution  function  of 
i  n+l  n  —  n 

the  time  until  the  next  transition  given  that  the  process  has  just  entered 
state  i,  then 


H At)  =  Z  p  F  (t)  =  I  Q  (t) . 
j«l  j*l  XJ 

We  shall  use  the  following  notation: 

"S’  ■  "g*  - 


(k)  (k)  ,  (k)  “  (k) 

Vij  =  Pijnlj  “  Wi  jf/ij  * 

For  convenience  we  write  p^  *  p^,  and  p^  =  pi> 

As  in  section  2.1  we  shall  assume  that  the  imbedded  Markov  chain  {X  } 

n 

is  irreducible  and  thus  has  a  stationary  probability  vector  j'. 

Let  be  the  time  for  a  first  passage  from  state  i  to  state  j  to  take 
place  in  the  Markov  renewal  process  and  let  G^j (t)  be  the  distribution  function 
of  If  m^j  *  /q t  dG^(t)  with  m^  *  m^  then  it  can  be  shown  (Lemma  2.1, 

Hunter  [  8  ])  that  for  r  _>  1 

(2.8)  -  jjplk<jr) + + pir)- 

so  that  if  M(r)  -  and  p(r)  ■  1 . 


^ — r""“~  ~~  ' '  *  '* 


(2.9)  (I  -  P)M(r)  =  Z  (r)P(r_s)[M(s)  -  M^S)]  +  P(r)E  -  PMjr), 

3-1  8  d  d 

In  particular  when  r  =  1  If  we  take  M^  -  M  we  have 


(2.10) 


(I  -  P)M  =  P(1)E  -  PM.. 

a 


Observe  that  when  the  Markov  renewal  process  {(X^.T^)}  degenerates  to  a 

Markov  chain  {X  }  by  taking  T  -  n  for  all  n,  y .  .  =  p..  and  y.  =  1.  For 
n  n  ij  ij  i 

such  a  case  equation  (2.10)  becomes  equation  (2.4).  Consequently  any 
general  technique  to  solve  equation  (2.4)  can  be  applied  to  the  more  general 
set  of  equation  (2.10).  Such  a  procedure  is  examined  in  section  5.2. 

For  each  fixed  j  £  S  the  instants  Tn  for  which  Xr  =  j  form  a  possibly 

delayed  renewal  process.  In  particular,  if  we  let  =  sup{n ^0|  Tn  <_  t}  and 
Nj(t)  =  number  of  n  (0  <  n  <  Nt)  such  that  Xr  =  j  then  the  vector 
(N^(t) ,N2(t) , • • • ,Nffl(t) }  forms  the  Markov  renewal  counting  process.  If  we 

are  given  that  Xq  =  i  then  we  can  define  N44(t)  =  N^(t).  Consequently  the 


ij 


j 


random  variables  N^(t)  are  analogous  to  the  occupation  time  random  variables 
in  the  Markov  chain  setting. 

Let  M±j(t)  =  ElN  (t)l  =  E[Nj(t)|XQ  =  i]  and  M(t)  -  [M^t)].  This 
matrix  function  is  also  called  the  Markov  renewal  kernel  and  can  be 
expressed  in  terms  of  semi-Markov  kernel,  ((inlar  [4  ]).  In  particular 

QO 

M(t)  =  [  z  Q.(”)(t)l, 
n=l  J 


where  Q^}(t)  -  P(Xn  -  j,  Tn  <  t|XQ  -  i),  (n  >  1), 

-  2  /jQ^"1)(t-y)dQik(y),  (n  >  2), 

keS 
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with  Qij  (t)  -  Q±  (t). 

In  Hunter  [  8 ]  it  was  shown  that 


(2.11) 


m(2) 

m.,<o  ■  +  (ji1-  -  r1)  +  »<1). 

13  "jj  2mjj  "u 


or  in  terms  of  matrices 

(2.12)  M(t)  =  tE(Hd)_1  +  |  E[(Md)"1)2M^2)  -  M(Hd)_1  +  o(l)E. 

(2) 

By  utilizing  general  expressions  for  H  and  Md  which  we  deduce  in  section  5.2, 
we  derive  a  collection  of  simple  expressions  for  equation  (2.12)  in  section  6.2. 

Associated  with  a  Markov  renewal  process  {(X^.T^)}  is  its  minimal  semi- 
Markov  process  (Y  }  (t  ^  0)  defined  as 

(where,  since  {Xn>  is  assumed  to  be  an  irreducible  Markov  chain,  sup  Tr  *  +  «*) . 
Thus  Y  may  be  regarded  as  the  state  that  the  Markov  renewal  process  is  in  at 
time  t. 

If  {X  }  is  irreducible  and  {(X  ,T  )}  is  aperiodic,  y.  <  00  for  all  i  e  S, 
n  n  n  i 

then  it  can  be  shown  (Cinlar  [ 4 ])  if  v'  ■  (v. , v-, • ■ • ,v  )  where 

*  i  z  in 


v  -  lim  P{Yt  -  j|x0  -  i}, 

J  t-*=° 


(2.13) 


ir'y  * 
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where  n'  is  the  stationary  probability  vector  of  the  Imbedded  Markov  chain, 

A  “  diag(p, ,p-, • • • ,p  )  and  p'  ■  (p, ,p0, *  *  * ,p  ).  Thus  the  limiting  distribution 
j.  l  n  **  i  z  in 

of  the  semi-Markov  process  can  be  found  easily  once  ir'  is  known. 


2.3.  Markov  chains  in  continuous  time 

When  the  semi-Markov  kernel  Q(t)  has  the  form 

(2.14)  (t)  -  PlJ[l  -  e-Xlt],  i,j  =  1,2, •• • »m,  t  >  0; 

with  p^  *  0  then  the  semi-Markov  process  associated  with  this  kernel  is 
a  Markov  process,  or  more  specifically  a  Markov  chain  in  continuous  time. 

We  shall  also  assume  that  0  <  <  «  so  that  the  process  is  stable  and 

regular. 

In  the  terminology  of  section  2.2  we  have  that 


Fij<t)  *  1  -  e~Xit>  1  *  J.  t  >  0, 


implying  that  for  l,j  e  S 


{1,2, ••*,»), 

(i  *  J);  “ij  *  ^  Vi 


Although  the  properties  of  the  process  are  determined  by  the  semi-Markov 
kernel  it  is  traditional  to  use  the  infinitesimal  generator  of  the  process 
when  examining  such  processes.  (inlar  [ 4  ]  shows  that  knowledge  of  Q(t)  as 
given  by  equations  (2.14)  is  equivalent  to  knowledge  of  the  infinitesimal 
generator  Q  *  [q^J.  In  particular 
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I-\  ,  i  "  j, 

Vij*  ljtJ* 


Conversely  =  -q 


and  p 


ij 


,  i  -  j. 


\  'qij/qil»  1  *  j- 


Observe  that  q^  0  for  i  ^  j, 
relationships  it  is  easy  to  see  that 


<  0  and  that  Eq 
j 


lj 


0 .  Prom  these 


(2.15)  I  -  P  =  (Qd)_1Q  and  A  =  -(Qd>  \ 

An  interesting  by-product  of  these  results  is  that  instead  of  using 
equation  (2.13)  to  find  v'  via  it’,  the  stationary  probability  vector  of  the 
irreducible  imbedded  Markov  chain  with  transition  matrix  P,  we  can  determine 
v'  from  the  infinitesimal  generator  of  the  process  by  solving  the  equations 


(2.16) 


v’Q  *  O'  subject  to  v'e  ■  1. 


Because  of  the  singularity  of  Q,  equation  (2.15)  lends  itself  to  treatment 
using  generalized  inverses  of  Q.  This  approach  is  considered  in  section  4.3. 

Expressions  for  the  mean  first  passage  times  and  the  moments  of  the  renewal 
counting  random  variables  can  be  obtained  from  the  relevant  Markov  renewal 
process  results  with  an  appropriate  Identification  of  parameters.  Further 
discussion  is  presented  in  section  5.3  and  6.3. 
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3.  Generalized  Inverses 
3.1.  Definitions 

In  section  2  we  presented  a  variety  of  applied  probability  situations 
where  we  have  a  set  of  linear  equations  involving  a  singular  matrix  that  we 
wish  to  solve.  The  simplest  class  of  "equation  solving  generalized  inverses" 
are  the  one  condition  generalized  inverses  which  we  shall  call  g- Inverses. 
Definition  3.1.:  A  g-inverse  of  a  matrix  A  is  any  matrix  A  such  that 

(3.1)  AA‘  A  -  A  . 

Generalized  inverses  are  in  general  not  unique  (unless  A  is  non-singular 
in  which  case  A  =  A  ^) .  By  imposing  additional  conditions  we  can  in  fact 
however  end  up  with  a  unique  generalized  inverse. 

+ 

Definition  3.2.:  The  Moore-Penrose  generalized  inverse  A  of  a  matrix  A  is 
the  (unique)  matrix  satisfying  the  conditions 

(i)  AA+A  =  A  (ii)  A+AA+  -  A+ 

(iii)  (AA+) '  -  AA+  (iv)  (A+A) '  -  A+A. 

We  have  made  no  statement  concerning  the  order  of  the  matrices  but  if  A 
is  a  rectangular  m  x  n  matrix  then  any  g-inverse  A  must  be  of  order  n  x  m 
so  that  the  matrix  multiplications  present  in  Definitions  3.1  and  3.2  are 
conformable.  In  the  application  areas  we  are  concerned  with,  only  square 
matrices  of  order  m  x  m  will  arise  and  for  such  matrices  there  is  another 
class  of  g-inverses  of  Interest. 

Definition  3.3.:  If  A  is  a  square  matrix  then  the  group  inverse  for  A,  if 
it  exists,  is  defined  to  be  the  matrix  A*  which  satisfies  the  conditions 


(i)  AA*A  =  A  (ii)  A*AA*  =*  A*  (iii)  AA*  -  A*A. 


2 

If  rank  (A)  *  rank  (A  )  then  a  group  inverse  exists  and  it  is  unique. 

There  is  an  extensive  literature  on  generalized  inverses  and  any  reader 
interested  in  getting  some  impression  as  to  the  richness  of  the  theory  that 
has  been  developed  should  consult  one  of  many  books  now  available  on  this 
subject.  For  example  Ben-Israel  and  Greville  [  1],  Bouillon  and  Odell  [2  ] 
and  Campbell  and  Meyer  [  3 ]  (this  latter  reference  contains  an  excellent 
chapter  on  applications  of  the  group  Inverse  to  finite  Markov  chains). 

3.2.  Solving  systems  of  linear  equations 

The  following  theorem  and  its  corollaries  provide  us  with  tools  for 
attacking  systems  of  linear  equations.  In  all  cases  we  assume  that  matrix 
multiplications  are  well  defined. 

Theorem  3.1. :  A  necessary  and  sufficient  condition  for  the  equation 
A  X  B  *  C  to  have  a  solution  is  that  AA  CB  B  *  C,  (the  condition  for 
consistency),  where  A  is  any  g-inverse  of  A  and  B  is  any  g-inverse  of  B. 

If  this  consistency  condition  is  satisfied  the  general  solution  is  given  by 
one  of  the  two  equivalent  forms;  either  (a): 

(3.2)  X  -  A~CB~  +  W  -  A-AWBB~ , 
where  W  is  an  arbitrary  matrix,  or  (b): 

(3.3)  X  -  A'CB"  +  (I  -  A"A)U  +  V(I  -  BB~)  , 
where  U  and  V  are  arbitrary  matrices. 

Proof ;  The  necessity  and  sufficiency  of  the  consistency  conditions  together 
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with  equation  (3.2)  can  be  found  in  Rao  [16]  (or  Theorem  6.3.2  of  Campbell 
and  Meyer  [3]).  The  equivalence  of  equations  (3.2)  and  (3.3)  follows  by 
taking  W  =  (I  -  A  A)U  +  V(I  -  BB  ),  or  conversely  taking 

U  *  V  -  +  A~AW  +  WBB").  D 

If  I  is  an  identity  matrix  I-  *  I  and  from  Theorem  3.1  we  deduce 
immediately  the  following  two  corollaries. 

Corollary  3.1.1.:  A  necessary  and  sufficient  condition  for  the  equation 
AX  ■  C  to  be  consistent  is  that  AA  C  ■  C,  where  A  is  any  g-inverse  of  A, 
in  which  case  the  general  solution  is  given  by 

(3.4)  X  -  A_C  +  (I  -  A~A)U  , 

where  U  is  an  arbitrary  matrix.  ° 

Corollary  3.1.2.:  A  necessary  and  sufficient  condition  for  the  equation 
XB  *  C  to  be  consistent  Is  that  CB  B  •  C,  where  B  is  any  g-inverse  of  A, 
in  which  case  the  general  solution  is  given  by 

(3.5)  X  -  CB_  +  V(I  -  BB")  , 

where  V  is  an  arbitrary  matrix.  ° 

Since  g-inverses  are  not  unique  let  us  write  A  {1}  for  the  set  of  one 
condition  g-inverses  A  of  A  satisfying  equation  (3.1).  Thus  if  X  e  A{1} 
then  A  X  A  ■  A.  Suppose  A  is  a  particular  g-inverse  of  A,  then  since 
AA  AA  A  ■  A  from  Theorem  3.1  we  can  deduce  that  the  equation  A  X  A  •  A  li 
consistent.  The  general  solution  of  this  equation  then  provides  us  with 
A{1}  given  A  .  In  fact  we  have  a  variety  of  characterisations  of  .>(1) 
given  a  particular  g-inverse  of  A. 
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Theorem  3.2,: 


If  A  is  any  g-inverse  of  A  then  all  g-inverses  of  A  can  be 


characterized  as  members  of  the  following  equivalent  sets: 

(3.6)  A{1)  *  {A  AA  +  W  -  A  AWAA  |  W  arbitrary), 

(3.7)  AU)  -  {a“aa“  +  (I  -  A~A)U  +  V(I  -  AA-)  |u,  V  arbitrary), 

(3.8)  A{1)  »  {A  +  H  -  A  AHAA  |H  arbitrary), 

(3.9)  A{1)  =  {A_  +  (I  -  A~A)F  +  G(I  -  AA-) |F,G  arbitrary). 

Proof :  The  characterizations  (3.6)  and  (3.7)  follow  from  equations  (3.2) 
and  (3.3)  with  C  =  B  =  A  and  the  equivalence  already  established  in  Theorem 
3.1.  Furthermore  by  taking  W  =  H  +  A  (3.6)  and  (3.8)  are  equivalent  as  are 
(3.7)  and  (3.9)  with  U  =  F  +  A-  and  V  =  G  +  |  A-. 

3.3.  Generalized  inverses  of  I  -  P 

The  following  theorem  provides  us  with  a  key  result,  hitherto  unreported 
in  the  literature. 

Theorem  3.3.:  Let  P  be  the  transition  matrix  of  a  finite  irreducible  discrete 
time  Markov  chain.  Let  u'  be  any  vector  such  that  u'e  t  0  and  let  t  be  any 
non-zero  vector.  Then 

(a)  I  -  P  +  tu'  is  non-singular. 

(b)  [I  -  P  +  tu']  ^  is  a  g-inverse  of  I  -  P. 

Proof :  (a)  For  any  matrix  X, 

det(X  +  tu')  -  det(X)  +  u’ (adj  X)t. 
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By  taking  X  •  I  -  P,  a  singular  matrix  we  have 


(3.10)  det(I  -  P  +  tu1)  -  u* [adj (I  -  P)]t. 

Furthermore  [adj (I  -P)](I  -  P)  -  (I  -  P)ladj(I  -  P)]  -  det(l  -  P)I  -  0, 
so  that  If  A  *  adj (I  -  P)  then 

(3.11)  AP  =  PA  =  A . 


If  P  Is  Irreducible  then  any  matrix  A  satisfying  equation  (3.11)  Is  a 

multiple  of  n  -  eir',  where  x'  Is  the  stationary  probability  vector  of  the 

Markov  chain.  Thus  adj(X  -  P)  **  kex*. 

Now  if  A  is  an  m  x  m  matrix  with  eigenvalues  y, ,y_, • • • ,p  and  if  y. 

x  /  m  i 

m 

is  a  zero  eigenvalue  then  tr(adj  A)  »  x  y..  Since  P  is  irreducible 

j*2  J 

its  eigenvalues  X. , X« ,  •  •  • ,  X  are  such  that  X,  -  1  is  the  only  eigenvalue 

1  Z  ID  X 

m 

equal  to  1.  Consequently  tr(adj  (1  -  P))  -  x  (1  -  X  )  i  0.  But 

j-2  J 

tr(adj  (I  -  P))  *  k  tr(ex' )  *  k  and  thus  from  equation  (3.10) 

det(I  -  P  +  tu')  *  k(u'e)(x't)  +  0, 

establishing  the  required  non-singularity. 

(b)  First  observe  that 


(I  -  P  +  tu')(l  -  P  +  tu?) 


.x-1 


I. 


so  that 


(3.12)  (I  -  P)(I  -  P  +  tu')"1  -  I  -  tu'(I  -  P  +  tu’)-1. 

Now  note,  using  equation  (2.2),  that 
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tt*  (I  -  P  +  tu')  ■  n'(I  -  P)  +  ir'tu'  “  (ir't)u'. 


and  thus 


(3.13) 


11  -1 
-  u' (i  -  p  +  tu')  . 

7T  t  **  *• 


Substitution  of  the  result  of  equation  (3.13)  into  equation  (3.12)  gives 


(3.14) 


-1  ” 

(1  -  P)(I  -  P  +  tu*)  "  1  -  pj  » 


and  hence  that 


(I  -  P)(I  -  P  +  tu')  (I  -  P)  -  I  -  P  -  ^  (I  -  P)  «  I  -  P, 

showing  that  (I  -  P  +  tu')  1  is  a  g-inverse  of  I  -  P  by  virtue  of  Definition  3.1.  □ 

Paige,  Styan  and  Wachter  f 15 J  developed  a  similar  result  in  which  they 

showed  that  I  -  P  +  eu'  is  non-singular  provided  u'e  /  0  and  that  its  inverse 

is  a  g-inverse  of  1  -  P.  Of  course,  this  is  a  special  case  of  Theorem  3.3 

with  t  -  e(j*  0).  However  it  is  the  generality  of  Theorem  3.3  that  will  prove 
useful  in  the  applications  to  follow. 

Corollary  3.3.1. :  Under  the  conditions  of  the  theorem,  any  g-inverse  of  I  -  P 
can  be  expressed  by  the  following  equivalent  forms: 


eu'H  Htir'  eu'Htn' 


(3.15) 


G  “  [I  -  P  +  tu'  ]  +  — ; —  +  — rz - 7 — r  w~ 

u'e  ir't  (u  e)  (ir 


where  H  is  an  arbitrary  matrix. 


.  eu'F  Gtir’ 

(3.16)  G  -  [I  -  P  +  tu*]"  +  ^  , 
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where  F  and  G  are  arbitrary  matrices. 

Proof :  Using  an  approach  similar  to  that  used  above 

(I  -  P  +  tu')_1(I  -  P)  «  I  -  (I-P  +  tu’)-1^' , 

and 


(I  -  P  +  tu*)e  *  (I  -  P)e  +  tu'e  =  (u'e)t  , 
with  (1  -  P)e  =  0  due  to  the  stochastic  nature  of  P,  implying  that 


(3.17) 


^  -  <1  -  P  ♦ 


and  consequently  that 

(3.18)  (I  -  P  +  tu')“1(I  -  P) 


I 


eu' 

u'e 

•» 


If  we  use  the  characterizations  (3.8)  and  (3.9)  then  equations 
(3.15)  and  (3.16)  follow  by  taking  A  m  I  -  P,  A  ■  (I  -  P  +  tu')  ^  and 
using  equations  (3.14)  and  (3.18). 

Corollary  3.3.2.:  Under  the  conditions  of  the  theorem  any  g-lnverse  of 
I  -  P  has  the  characterization 

(3.19)  G  =  [I  -  P  +  tu' j”1  +  ef'  +  g*’. 


for  arbitrary  f  and  g. 

Proof :  Equations  (3.16)  and  (3.19)  are  equivalent  by  taking  F  ■  ef'  and 
G  ■  gir'  or  conversely  by  taking 
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f 


u'F 

— 71  and  g 
u  e  “ 


Gt 

irrt 


Similarly  equations  (3.15)  and  (3.19)  are  equivalent  by  taking 
H  *  ef'  +  gir'  or  conversely  by  taking,  for  example. 


f' 


y’H 

^  and  g  =  [I 


eu’  Ht 
u'e^  n't 


□ 


A  variety  of  generalized  inverses  of  I  -  P  have  appeared  in  the 
literature  when  their  derivation  was  often  obtained  by  'ad  hoc'  techniques. 
In  the  following  corollary  we  exhibit  their  representation  in  terms  of  the 
characterization  given  by  (3.19). 

Corollary  3.3.3.:  Let  P  be  the  transition  matrix  of  a  finite  irreducible 
Markov  chain  with  stationary  probability  vector  x'  and  IT  =  ex' .  The 
following  are  all  generalized  inverses  of  I  -  P. 


(a) 

(b) 

(c) 

(d) 

(e) 


(f) 


Gl- 


G2  “ 


G3 


If  P 


[i  -  p  +  n]  x. 

[I  -  P  +  eu' ]  ^  provided  u'e  j  0. 

[I  -  P  +  n]"1  -  n. 

-1  -1/2 
[I  -  P  +  axe']  -  all  where  a  *  (mx'x) 

[I  -  P]~  +  B  adj(I  -  P)  for  any  (I  -  P)"  and  B  J4  0. 


then  I  -  P  has  a  g-inverse  of  the  form 


B 
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where  u'  -  (0',1),  t'  -  (0\1),  f'  -  -(8’  (I  -  Pu)-1,l). 

Proof :  (a)  follows  by  taking  t  -  e,  u  *  ir,  f  *  0,  g  ■  0.  was  first 

recognized  as  a  g-inverse  of  I  -  P  by  Hunter  [  8  ]  and  Is  known  as  the 

"fundamental  matrix"  of  the  Irreducible  transition  matrix  P  (Kemeny  and 
Snell  [  9 ]). 

(b)  As  observed  earlier,  G2  was  Identified  as  a  g-lnverse  of  I  -  P 
by  Paige,  Styan  and  Wachter  [15]. 

(c)  G^  follows  by  taking  t  *  e,  u  *  ir,  f  =  -tt,  g  ■  0,  and  is,  in  fact, 
the  group  inverse  (I  -  P)  (see  Definition  3.3)  as  first  shown  by  Meyer  [11]; 
(see  also  Campbell  and  Meyer  [3]). 

(d)  G^  follows  by  taking  t  -  ir,  u  ■  ae,  f  ■  -air,  g  ■  0,  and  was 

shown  by  Paige,  Styan  and  Wachter  [15]  to  be  the  Mbore-Penrose  generalized 

inverse  of  I  -  P,  (I  -  P)+,  (see  Definition  3.2). 

(e)  From  the  proof  of  the  non- singularity  of  I  -  P  +  tu'  we  saw 
that  adj(I  -  P)  =  keir'  with  k  +  0. 


I  -  P  -a 

11  “ 

A11 

A12 

(f)  I  -  P  +  tu'  ■= 

- 

"B?  2‘Pmm 

_A21 

A22 

where  t'  **  (0',1)  4  O'  and  u'e  “  (O'  ,l)e  ■  1  fO.  Thus  I  -  P  +  tu'  is  non¬ 
singular  and  from  the  results  concerning  inverses  of  partitioned  matrices. 


[I  -  p  +  tu*]"1 


11 

B12 

21 

B22 

where  B22  •  (A22  -  A2lAH  *Ai2^  1  “  1' 


B21  *  ~B22A21A11 


-1 


§'  (!  -  Pu)_1, 


B12  "  “All  A12B22 


(I  -  Pn)"1a, 


B11  "  A11  '  A11  A12B21 


(I  -  Pn)"1  +  (I  -  Pn)'1  ag'd  -  Pu)-1. 


In  determining  these  expressions  we  have  made  use  of  the  fact  that 

(I  -  P)e  -  0  so  that  1  -  p  *  3'e  and  (I  -  P,.)e  =  a  implying  that  1-p 

~  -  mm  ~  ~  n  ~  mm 

=  3’  (I  -  P^)  ^a.  The  required  form  now  follows.  This  result,  without  the 
particular  representation  of  the  form  (3.19),  was  reported  by  Rohde  [17] 
as  being  provided  by  a  personal  communication  from  J.  Hearon.  (An  expression 
for  the  group  inverse  of  I  -  P  with  P  so  partitioned  has  been  given  by 
Campbell  and  Meyer  [  3 ] ) . 


3. A.  Generalized  inverses  of  Q 

If  Q  »  is  the  infinitesimal  generator  of  the  continuous  time  Markov 

chain  and  P  is  the  transition  matrix  of  the  imbedded  discrete  time  Markov 
chain  then  from  equations  (2.15)  it  is  easy  to  see  that 

(3.20)  Q  -  -  A-1(I  -  P). 

If  P  is  irreducible  we  have  that  I  -  P  has  rank  m  -  1  and  that  P  has  a 
simple  unit  eigenvalue.  The  equivalence  of  these  two  statements  follows 


since  P  is  stochastic  (pg.  133,  Marcus  and  Mine  [10]).  However,  even  though 
Q  is  not  stochastic  we  can  establish  analogous  results. 

Lemma  3.4.:  If  P  is  irreducible  then  for  Q  given  by  (3.20) 

a)  Q  has  rank  m  -  1, 

b)  Q  has  a  simple  zero  eigenvalue. 

Proof :  Since  I  -  P  has  rank  m  -  1  and  Q  is  derived  from  I  -  P  by  a  non¬ 
singular  transformation,  Q  also  has  rank  m  -  1. 

Since  det(Q)  =  0  it  is  easy  to  see  that  Q  has  at  least  one  zero  eigenvalue 
but  to  establish  the  simplicity  of  this  eigenvalue  requires  more  care.  We  make 
use  of  the  result  that  zero  is  a  simple  eigenvalue  of  Q  if  and  only  if  det(Q)  *  0 
and  tr(adj  Q)  0  (cf.,  p.  218,  Mirsky  [13]). 

Now  from  equation  (3.20), 

adj  Q  =  adj (I  -  P)  adj(-A_1), 

=  keii'  (-l)m-1(  “  A . )  A, 
i=l  1 

where,  from  the  proof  of  Theorem  3.3  k  ^  0  and  from  the  specifications  in 

m  111 

section  2.3,  n  A.  ^  0  with  A  =  diag (-r— ,—,•••  ,-r-) .  Thus 

1=1  1  X1  X2  \n 


(3.21)  adj  Q  *  cerr’A  with  c  i1  0, 

m 

implying  tr(adj  Q)  =  cir'A  e  =*  c  2  t—  ^  0. 

i*lAi 


Q 


Armed  with  these  observations  we  can  now  generalize  Theorem  3.3. 

Theorem  3.5.:  Let  Q  be  the  infinitesimal  generator  of  a  continuous  time  Markov 


chain  with  irreducible  imbedded  discrete  time  jump  chain.  Let  u'  be  any  vector 


such  that  u'e  j*  0  and  let  t  be  any  non-zero  vector.  Then 

(a)  Q  +  tu'  is  non-singular. 

m  mt 

(b)  (Q  +  tu')  ^  is  a  g-inverse  of  Q. 

Proof:  (a)  Following  the  proof  of  Theorem  3.3  (a)  we  have  that 


det  (Q  +  tu’ )  =  u’  adj(Q)t  , 


=  c(u’e) (j’ At)  t  0, 
(b)  With  minor  changes  we  can  use  the  arguments 
proof  of  Theorem  3.3  (b).  If  I  -  P  is  replaced  by  Q  and 


using  (3.21). 
given  in  the 


ir*  by  v' 


(« 


so  that  v'Q  *  O',  then  the  same  procedure  gives 


Q(Q  +  tu1)"1  Q  =  Q, 


and  hence  the  required  conclusion.  □ 

Without  going  into  all  the  details  we  can  work  through  the  relevant 
corollaries  of  Theorem  3.3  to  conclude  that ,  under  the  conditions  of  Theorem 
3,5,  any  g-inverse  of  Q  has  the  characterization 

(3.22)  H  ■  (Q  +  tu’)  ^  +  ef'  +  gv'. 

The  results  of  Theorem  3.5  and  expression  (3.22)  are  new.  To  my  knowledge 
the  theory  generalized  inverses  has  not  previously  been  applied  to  infinitesimal 
generators  of  Markov  processes. 


4.  Stationary  Distributions 

4.1.  Markov  chains  in  discrete  time 

Using  the  theory  of  generalized  Inverses  we  can  now  obtain  a  general 
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vanwj*  jjr  - 


expression  for  the  stationary  probability  vector  x'. 

Theorem  4.1. :  Let  P  be  the  transition  matrix  of  a  finite  irreducible  Markov 
chain.  If  (I  -  P)  is  any  g-inverse  of  I  -  P  and  if  A  =  I  -  (I  -  P) (I  -  P) 
then 


(4.1) 


y'A 
v' Ae 


» 


where  y'  is  any  vector  such  that  v'Ae  +  0. 

Proof.  Observe  that  equation  (2.2),  x*(I  -  P)  =  0',  has  the  form  XB  *  C  where 
X  =  x’ ,  B  =  I  -  P,  C  =■  O’.  Using  Corollary  3.1.2  we  see  that  the  consistency 
conditions  are  obviously  satisfied  and  that  equation  (3.5)  implies  that  the 
general  solution  is  given  by 


x'  «  z'(I  -  (I  -  P)(I  -  P)  )  =  z'A, 


where  z'  must  be  chosen  so  that  x'e  *  1  *  z'Ae. 

Let  v'  be  any  vector  such  that  v'Ae  ^  0  and  take  z'  =  v'/v'Ae  (^  O'). 
Then,  for  such  a  choice,  x'  *  v'A/v’Ae  and  x’e  *  1. 

There  are  a  variety  of  ways  that  we  can  use  Theorem  4.1.  Firstly, 
suppose  we  are  given  a  computer  subroutine  for  generating  g-inverses.  Can 
we  use  this  package?  In  order  that  we  can  use  equation  (4.1)  we  have  to  be 
sure  that  we  can  in  fact  find  a  suitable  v' .  The  following  corollary 
establishes  the  required  verification  providing  us  with  an  affirmative 
answer  to  the  query. 

Corollary  4.1.1.;  Under  the  conditions  of  the  theorem,  if  (I  -  P)  is  any 
g-lnverse  of  I  -  P  then  Ae  ^  0  and  thus  we  can  always  find  a  v'  such  that 
v'Ae  i*  0. 
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Proof.  Since  any  g-inverse  of  I  -  P  can  be  characterized  by  the  form  given 
by  equation  (3.19)  ve  see  that 

A  *  I  -  (I  -  P)[(I  -  P  +  tu')-1  +  ef' 
t 

■  ["^5“  -  (I  -  P)g]7' ,  using  equation  (3.14). 
t 

Thus,  since  i'e  *  1,  Ae  *  -  (I  -  P)g.  Now  suppose  that  Ae  ■  0.  This  then 

implies  that 

Yi  -  «  - p>?  - 

7' t 

so  that  1  ■  ~7~~  =  n'  (I  -  P)g  =  0,  a  contradiction,  and  thus  Ae  t  0.  □ 

TT  t  -  ~  " 

Another  way  to  use  Theorem  4.1  is  to  take  a  particular  form  of  g-inverse 

of  I  -  P  that  does  not  directly  involve  knowledge  of  j  and  use  such  a  g-inverse 

in  the  procedure  described  by  equation  (4.1). 

Corollary  4.1.2. :  If  we  take 

(I  -  P)“  -  (I  -  P  +  tu')"1  +  ef’, 

where  t  j*  0,  u'  such  that  u'e  +  0,  and  f  are  arbitrarily  chosen  then 

u'  [I  -  P  +  tu']"1 

(4.2)  »•  -  - - —  rr  * 

u' [I  -  P  +  tu' ]  e 
Proof:  With  (I  -  P)-  so  chosen 

A  -  1  -  (I  -  P){(I  -  P  +  tu')"1  +  ef'), 

■  tu* (I  -  P  +  tu')  \  using  equation  (3.12). 


Substitution  in  equation  (4.1)  yields 


v' tu' (I  -  P  +  tu' ) 

I*  "  '  _i  • 

v ' tu  (I  -  P  +  tu')  e 

Equation  (4.2)  now  follows  by  dividing  the  numerator  and  denominator  by  v't 
(since  we  can  obviously  find  a  v'  so  that  v't  t  0).  Note  also  that  by  equation 
(3.13)  the  denominator  u'(I  -  P  +  tu')  ^e  is  non-zero.  n 

~  « «v  *• 

Equation  (4.2)  is  a  new  result  and  gives  a  very  general  procedure  for 
finding  it*.  Further  studies  need  to  be  carried  out  to  determine  efficient 
choices  of  t  and  u  to  simplify  the  computation. 

The  generality  of  the  procedure  outlined  in  Theorem  4.1  is  such  that  all 
known  explicit  methods  for  finding  stationary  distributions  of  finite  irreducible 
Markov  chains  can  be  expressed  in  this  framework.  We  show  how  some  of  the 
more  well  known  techniques  can  be  deduced  from  this  theorem. 

Corollary  4.1. 3.:  Let  P  be  the  transition  matrix  of  a  finite  m-state 
Irreducible  Markov  chain. 

(a)  For  any  u  such  that  u'e  t  0, 

(4.3)  r '  =  u'[I  -  P  +  eu'] 

(b)  Let  (1  -  P)  be  a  generalized  inverse  of  I  -  P  such  that 
e'[I  -  (I  -  P)(I  -  P)"]e  +  0,  then 

e'[I  -  (I  -  P)(I  -  P)"] 

(4.4)  n'-  - -  . 

e' [1  -  (I  -  P)(I  -  P)~]e 
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(c)  Let  Dj  be  the  determinant  of  the  matrix  formed  by  removing  the  jth  row 
and  jth  column  from  I  -  P,  then 

e’[adj(I  -  P)]d 
=  tr(adj(I  -  P))~  * 

(4.5) 

(  s  D  ) 

j=l  J 


(d)  The  matrix  (I  -  P)^  ■  I  -  P  +  t^e^'  where  is  the  jth  elementary  vector 
and  tj  ■  e  -  (I  -  p)ej»  which  can  be  formed  from  I  -  P  by  replacing  its  jth 
column  by  e  is  non-singular;  its  inverse  is  a  g- inverse  of  I  -  P  and 


(4.6) 


w'  -  •j’KI  "  DjT1,  (j  *  1»2,  *  *  ’  ,m). 


(e)  If  P  » 


then 


<§'  (I  -  Pn)_1,  1) 

(4.7)  n' - - -  . 

(g'(I  -  Pu)  .  D? 


Proof : 

(a)  In  equation  (4.2)  take  t  -  e  and  (4.3)  follows  by  observing  that 
u’(I  -  P  +  eu')  ^"e  *  1,  which  can  be  established  from  equation  (3.13)  by 


po8tmultiplying  by  t  *  e.  This  particular  result,  with  a  direct  proof,  was 
given  earlier  by  Paige,  Styan  and  Wachter  [15]. 


(b)  This  follows  immediately  from  equation  (4.1)  with  v  ■  e.  [Decell  and  Odell 
[5  ]  derived  this  result  under  the  additional  assumption  that  (1  -  P)(I  -  P) 
should  be  syzmetric.  This  rather  severe  restriction  establishes  the  non-zero 
nature  of  k  *  e'Ae.  Since,  if  a  •  Ae  then  a'  »  e'A  and  it  follows  upon 
simplification  that  a'ot  =■  k.  Now  a'a  >  0  iff  a  )>  0,  so  that  k  ■  0  only  if 

a  ■  0  but  this  would  imply  that  e  *  (I  -  P)(I  -  P)  e  and  hence  that 
ii'e  ■  u'  (1  -  P)  (I  -  P)  e  *  0,  a  contradiction.] 

(c)  In  Theorem  4.1  take  v'  *  e^’[adj(I  —  P) ] .  Then 
v’A  -  e^tadjd  -  P)][I  -  (I  -  P)(I  -  P)"], 

-  e1' [adj (I  -  P)],  since  [adj (I  -  P)](I  -  P)  -  0, 

-  v\ 

so  that 

v'A  e' [adj(I-P)] 

ir*  -  — rr—  *  -  . 

Y  ~  e^'  [adj  (I  -  P)]e 

Now  from  the  proof  of  Theorem  3.3  we  have  that  adj  (I  -  P)  ■  kH  »  [kir^  ]  where 
k  ■  tr(adj(I  -  P))  +  0.  Also  adj  (I  -  P)  ■  [a^]  w*1®1®  is  th®  cofactor  of 
the  (i,j)th  element  of  I  -  P.  Thus  “  kir^  ■  “  D j ’  Consequently, 

®^'  t®dj (I  -  P)]  -  ^ail,021’  *  *  ’ * °ml^  *  ^all,022*”’’amm^» 

-  e'  [adj (I  -  P)]d  -  (D1,D2,"*,Dm), 

and 


27 


e’[adj(I  -  P)]e 


tr(adj (I  -  P)), 


m 

*  2  01,  j 

j-i  a 


=■  E  D,  (=  k  E  ir,  «=  k  +  0) , 

j-1  J  j-1  J 


and  equation  (4.5)  follows. 

This  representation  of  tt *  in  terms  of  was  originally  due  to  Mihoc; 
(cf.  Frechet  [7],  Hunter  18]). 

(d)  The  elements  of  the  jth  column  of  a  matrix  can  be  reduced  to  zero  by 
port  multiplication  by  I  -  '  and  thus 

(I  -  P)j  -  (I  -  «  e  •>  +  ee^. 


1  -  F  +  Sjffj1 


Observe  that  t  ^  0  and  that  e^e  *  1  which,  form  Theorem  3.3  establishes 
the  non-singularity  and  g-inverse  properties  of  (I  -  P)  ^ .  Expression  (4.6) 
follows  from  equation  (4.2)  upon  noting  that  equation  (3.13)  implies 


e^ '  [I  -  P  +  (Pe^  ~  ej  +  e)ej']_1e  *  it'e/Tt' (Pe^  -  e^  +  e)  -  1. 

This  particular  procedure  was  also  suggested  by  Paige,  Styan  and  Wachter  [15 ]• 
(e)  With  (I  -  P)  as  given  by  Corollary  3.3.3(f) 


A  -  I  -  (I  -  P)  (I  -  P)  - 


0  9 

§’  (I  -  P^"1  1 


Thus  using  equation  (4.1)  with  v  -  e  we  obtain 
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e'A  <{*'  (I  -  *u>“1.  1) 

7  =  p'  Ae  =  IT  • 

-  -  (6*  (I  -  Pu)  ,  l)e 

This  result  was  reported  by  Rohde  [17].  An  alternative  proof  of  this 
result  via  the  group  inverse  (I  -  P)^  is  given  by  Meyer  [11]. 

An  alternative  derivation  can  also  be  given  by  using  equation  (4.6) 
with  j  *  m",  since  with  P  so  partitioned 

’I-P11  s 

-  p>m  - 

-§'  1 

(x  -  p^-1  -  a(i  -  Pj^rW  (i  -  pu)-1  -4d  - 

id  -  p>  r 1  - 

®  -i 

46*  (I  -  P^)  A  4 

where  4  -  (1  +  S' (I  -  P^)  *e)  .  Equation  (4.7)  now  follows  from  equation 
(4.6).  This  technique  was  used  by  Meyer  [12]  and  also  by  Snell  [18].  a 

If  expression  (4.2)  is  used  as  the  basis  for  a  procedure  to  determine  it' 
then  there  is  still  considerable  flexibility  in  the  choice  of  u'  and  t.  Paige, 
Styan  and  Wachter  [15]  carried  out  an  error  analysis  and  computational  comparison 
between  a  variety  of  algorithms  that  included  the  techniques  specified  by 
equations  (4.3)  (4.4)  and  (4.6)  together  with  rank  reduction,  limits  of  matrix 
powers  and  least  squares  procedures.  Their  study  concluded  with  a  recommendation 
for  the  method  given  by  equation  (4.3)  with  u'  =  e^’P  (j  ■  m  for  convenience) 
using  Gaussian  elimination  with  pivoting  to  solve  the  equation  ir*(I  -  P  +  eu')  * 
This  procedure  gave  the  fastest  computing  times  and  the  smallest  average 
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residual  errors. 


4.2.  Semi-Markov  processes 

From  equation  (2.13)  we  see  that,  under  certain  conditions,  the  limiting 
distribution  of  a  semi -Markov  process  can  be  expressed  in  terms  of  ir',  the 
stationary  probability  vector  of  the  underlying  Markov  chain.  Obviously  we 
can  use  any  of  the  techniques  of  Theorem  4.1  and  its  corollaries  to  find  ir' 
and  then  compute  v' .  However  are  there  any  procedures  that  will  give  us 
some  simple  structure  to  the  algorithm?  The  following  theorem  addresses 
Itself  to  these  questions. 

Theorem  4.2.:  Let  {(X  ,T  )}  be  an  aperiodic  finite  Markov  renewal  process  with 
-  n  n 

the  Markov  chain  {X  }  having  an  irreducible  transition  matrix  P.  Let  (u. } 
n  i 

be  the  mean  holding  times  in  the  states  with  A  »  diag(u1,p2* *  * ’ »Mm)  and 
(j  »  Ae.  Let  v'  *  (v1,vot**»,v  )  be  the  vector  of  limiting  probabilities  for 
the  minimal  semi-Markov  process  then  for  any  t  ft  0  and  u  such  that  u'e  ft  0 

u’ [I  -  P  +  tu’ ]-1A 

(4.8)  v’  -  - - - - r  . 

u’[I  -  P  +  tu’]  y 

In  particular, 

(4.9)  y'  •  u* ll  -  P  +  yu']  *A  . 

Proof :  Equation  (4.8)  follows  from  equation  (2.13)  and  equation  (4.2),  with 
cancellation  from  numerator  and  denominator  by  u' [I  +  P  +  tu']  *e. 

Equation  (4.9)  now  follows  by  taking  advantage  of  equation  (3.13)  with 


Observe  that  [I  -  P  -  yu']  ^  is  a  g-inverse  of  I  -  P  which  reduces  to 
[I  -  P  +  eu']  *  when  the  Markov  renewal  process  degenerates  to  a  Markov 
chain  with  y  ■  e.  Thus  equation  (4.9)  is  the  analog  of  equation  (4.3)  in  this 
more  general  context. 

Note  also  that  there  is  a  one  to  one  correspondence  between  v'  and  ir' 
so  that  in  any  application  knowledge  of  either  suffices.  It  is  easily  seen 
that 

it* A  v* A  * 

y '  ■  is  equivalent  to  ir'  *  - . 


4.3.  Markov  chains  in  continuous  time 

Since  Markov  chains  in  continuous  time  are  special  semi-Markov  processes 
we  can  use  the  results  of  the  previous  section  with  A  -  and  y  *  -Qd*e. 

Furthermore,  instead  of  expressing  our  algorithm  in  terms  of  P,  the  transition 
matrix  of  the  imbedded  jump  chain  we  can  give  alternative  expressions  using 
Q,  the  infinitesimal  generator  of  the  process.  This  leads  immediately  to  the 
following  theorem. 

Theorem  4.3.:  Let  t  +  0  and  u  be  vectors,  with  u'e  #  0  then  the  stationary 
probability  vector  y'  for  the  Markov  chain  in  continuous  time  with  imbedded 
jump  chain  transition  matrix  P  can  be  expressed  by 

u*  [I  -  P  +  tu'rV]1 

(4.10)  v’  «- - — - r-V  . 

u'[I  -  P  +  tu']  xQd  e 

In  particular, 
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(4.11) 


v*  -  u'[I  -  P  +  Q^eu']"^1  • 

Alternatively,  v'  can  be  expressed  in  terms  of  the  infinitesimal  generator 
as 

u*  [Q  +  tu'f1 

(4.12)  v’  -  - - -r-  . 

u'  (Q  +  tu'  ]  e 

In  particular, 

(4.13)  v'  -  u'[Q  +  eu']  *. 

*•  *v  ■*» 

Proof :  Equations  (4.10)  and  (4.11)  follow  from  equations  (4.8)  and  (4.9) 
after  substitution  for  A  and  p  and  replacing  both  t  and  u  by  -t  and  -u 
respectively. 

Equations  (4.12)  and  (4.13)  now  follow  from  equation  (4.10)  and  (4.11) 
with  I  -  P  *  Q^Q  ami  t  replaced  by  Q^t  1*0.  ° 

An  interesting  observation  is  that  equations  (4.12)  and  (4.13)  can  also 
be  obtained  directly  by  solving  the  equations  v'Q  *  0  subject  to  v'e  *  1  using 
the  g- inverse  approach  to  solving  systems  of  linear  equations  as  outlined  in 
section  4.1  but  using  the  g-inverse  of  Q  as  found  in  section  3.4. 

Although  we  won’t  restate  the  equivalent  theorems  in  this  section,  every 
result  derived  in  section  4.1  follows  for  Markov  chains  in  continuous  time  but 
with  I  -  P  replaced  by  Q  and  it'  replaced  by  v'.  This,  of  course,  is  a  direct 
consequence  of  the  characterization  of  g- inverses  for  Q  as  given  by  equation 
(3.20). 


5.  Mean  First  Passage  Time  Matr  es 
5.1.  Markov  chains  in  discrete  time 

Let  M  -  [ j  3  he  the  mean  first  passage  matrix  for  a  Markov  chain  with 
Irreducible  transition  matrix  P  and  stationary  probability  vector  Let 

II  ■  ex' .  Then  from  equation  (2.4) 

(I  -  P)M  -  E  -  PMd  , 

where,  from  equation  (2.5)  Md  *  (nd) 

Observe  that  equation  (2.4)  is  of  the  form  AX  •  C  where  A  ■  I  -  P, 

X  -  M  and  C  *  E  -  P(JId)  \  The  general  procedure  for  solving  this  system 
of  linear  equations  is  given  by  Corollary  3.1.1  and  by  taking  a  general 
form  for  the  g- inverse  of  I  -  P  we  are  led  to  the  following  result. 

Theorem  5.1. :  If  G  is  any  g- inverse  of  I  -  P,  then 

(5.1)  M  =  [GH  -  E(GH)d  +  I  -  G  +  EGd]D  , 
where  D  =  (nd) 

Proof :  By  equation  (3.4)  the  general  solution  to  equation  (2.4)  is  given  by 

(5.2)  M  -  6(1  -  PD)  +  {I  -  G(I  -  P)}U, 

where  U  is  an  arbitrary  matrix,  provided  the  consistency  condition 

[I  -  (I  -  P)G](E  -  PD)  =  0  , 

is  satisfied.  With  G  taken  in  the  general  form  as  given  by  equation  (3.19), 
i.e.,  for  suitable  t,  u,  f  and  § 
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G  =  [I  -  P  +  tu'  ]-1  +  ef'  +  gir', 
we  have,  using  equation  (3.14)  that 

t 

[I  -  (I  -  P)G](E  -  PD)  =  [~  -  (I  -  P)g]*'  [ee’  -  PD] 

t 

=  "  (I  “  p)8He'  -  tt'D] 

*  0  since  ir*  D  =  e' , 

showing  that  equations  (2.4)  are  in  fact  consistent.  Now,  using  equation 
(3.18), 

u' 

[I  -  G(I  -  P)]U  =  e[^-  -  f *  (I  -  P)]U, 

=  eh’U  5  eb',  say. 


Thus  equation  (5.2)  can  be  written  as 

(5.3)  M  -  G(E  -  PD)  +  eb'  . 

2 

Observe  that  the  m  arbitrary  elements  of  U  have  been  reduced  to  only  m, 
the  elements  of  b1.  These  can  be  determined  explicitly  due  to  the  restriction 
that  Md  -  (nd)  1.  Suppose  b'  *  ^bi,b2» '  ”  »bm^  and  let  B  *  dla8(b1»b2» ’ * ’ *bm)  • 
From  equation  (5.3)  forming  the  matrices  of  diagonal  elements  gives 

D  -  (Gn)dD  -  (GP)dD  +  B, 

implying  that 

(5.4)  B  -  (I  -  (Gn)d  +  (GP)d]D. 
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Since  eb'  =  EB  and  E  =  IID  substitution  of  equation  (5.4)  into  equation  (5.3) 
yields 

(5.5)  M  =  [Gn  -  E(Gn)J  -  GP  +  E(GP)  .  +  E]D  . 

a  a 

Further  simplification  of  equation  (5.5)  is  possible  since  we  have 
shown  above  that 


I  -  G  +  GP  =  eh'  , 

and  thus 


E(I  -  G  +  GP),  =  ee’(eh').  «  eh'. 

d  ~~  —  a  — 

Hence 

E  -  EGd  +  E(GP)d  -  I  -  G  +  GP  , 

implying  that  E  +  E(GP)d  -  GP  =  I  -  G  +  EGj.  Substitution  into  equation  (5.5) 
gives  the  required  result,  equation  (5.1). 

Theorem  5.1  presents  a  new  result.  It  has  the  same  desirable  property 
alluded  to  earlier,  following  Theorem  4.1,  namely  that  any  computer  package 
that  generates  g-inverses  can  be  used  to  determine  means  of  the  first 
passage  time  distributions. 

Because  we  have  used  an  arbitrary  g-inverse  of  I  -  P  in  developing 
Theorem  5.1  we  have  considerable  flexibility  in  choosing  a  particular  g-inverse. 
Firstly  the  form  of  the  solution  for  H  given  by  equation  (5.1)  does  not 
depend  on  the  choice  of  f  and  g. 

Corollary  5.1.1. ;  If  G  -  G^  +  ef'  +  gir'  where 
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Gq  *  [I  -  P  +  tu' ]-1  (t  i  0,  u'e  ^  0), 

then 

(5.6)  M  -  [GQn  -  E(G0II)d  +  I  -  GQ  +  E(GQ)d]D  , 
where  D  -  (Hd>  1. 

Proof:  Let  G  *  Gq  +  H  where  H  ■  ef '  -t-  gir1 .  Then 

Hn  *  (f'e)n  +  gw’  =  (f'e)n  +  H  -  ef'  , 

and 

E(Hn)d  *  (f'e)End  +  EHd  -  E(ef')d  -  (f'e)Il  +  EHd  -  ef’. 

Consequently  HII  -  E(HII)d  *  H  -  EHd  and  the  result  follows  by  substitution 
in  equation  (5.1).  O 

The  advantage  of  equation  (5.6)  is  that  any  computation  with  a  more  general 
g-inverse  is  effectively  the  same  as  that  performed  by  taking  a  g-inverse  of 
the  form  GQ  =  [I  -  P  +  tu']”1'. 

However  by  placing  some  additional  restrictions  on  the  form  of  the 
g-inverse  we  can  simplify  the  form  of  the  expression  for  M. 

Corollary  5.1.2.:  (a)  If  G  ■  [I  -  P  +  eu' ]  ^  +  ef'  then 

(5.7)  M  *  [I  -  G  +  EGd]D. 

(b)  If  G  ■  [I  -  P  +  eu*]  ^  +  ef*  +  gir'  then 

(5.8)  M  -  (I  -  Gq  +  E(GQ)d]D  , 
where  Gq  ■  [I  -  P  +  eu’ ]  ^  and  D  -  (nd) 
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Proof:  (a)  With  G  as  specified 


GII  =*  [I  -  P  +  eu*  +  ef'ev'  , 

ej* 

*  -=4—  +  (fejeir'  ,  using  equation  (3.15), 
u  e  ~ 

*  where  $  =>  (1/u'e)  +  f'e. 

Thus  E(Gn)d  =  $End  =  gn  =  Gn  and  the  result  follows  from  equation  (5.1). 

(b)  With  G  as  specified,  part  (a)  with  f  =  0  implies  that 

E(G  n) ,  =  G„  and  the  result  follows  from  equation  (5.2).  D 

'  0  d  0 

Special  cases  of  Corollary  5.1.2  have  appeared  in  the  literature.  In 
particular  when  f’  =  O'  and  u'  =  n’  equation  (5.7)  becomes 

M  m  [I  -  Z  +  EZd]D  , 

where  Z  -  [I  -  P  +I!]"1,  the  fundamental  matrix  of  the  irreducible  Markov 
chain,  as  obtained  by  Kemeny  and  Snell  t  9  ]. 

Similarly  when  £'  *  — -ft*  and  u'  =  *’  equation  (5.7)  becomes 

M  =  [I  -  T  +  ETdJD  , 

where  T  *  [i  -  P  -n] -  II,  the  group  inverse  of  I  -  P;  as  obtained  by  Meyer  Ill]. 

There  are  some  computational  considerations  that  need  to  be  taken  into 
account.  Both  Z  and  T  effectively  require  the  prior  determination  of  x’ 
which  of  course,  requires  the  use  of  some  generalized  inverse  of  I  -  P.  Why 
not  use  the  same  g- inverse  in  one  of  the  forms  (5.1),  (5.7)  or  (5.8)?  In  the 
corollary  that  follows  we  make  use  of  some  of  the  g-inverses  used  in  establishing 
Corollary  4.1.3. 
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then 


Corollary  5.1.3.; 

(a)  If  6  -  [I  -  P  +  eu’]-1, 

(5.9)  M  -  [I  -  G  +  EGd][eu'G)d]~1. 

(b)  If  G  ■  [(I  -  P)j]-1  s  f8ij]  then, 

(5.10)  M  -  [I  -  G  +  EGd]D  +  (e^e'  -  ee^'). 

where  D  -  diag(  . 

gjl  gj2  gja 


(c)  If  P 


11 

8'  p 


nun 


a  2  (I  -  Pi!)"1?,  b*  =§»(!-  Pu)_1,  A  s  1  +  b’e  , 


then 

(5.11)  M 


[ab'  -  E(ab')d  +  A{I  -  (I  -  Pu>  1  +  E((I  -  Pn)  1) d> ] ( (eb * ) d) 
[Ae'((I  -  P11)"1)d  -  e'(ab,)d]((eb')d)'1 


Proof: 

(a)  Equation  (4.3)  gives  f'  ■  u’G  and  the  result  follows  from  equation  (5.8) 
by  noting  that  JId  *  (ex'  )d  ■  (eu'G)d. 

(b)  From  Corollary  4.1.3  we  see  that  G  ■  [I  -  P  +  ^  where 

tj  -  e  -  (I  -  P)ej.  Furthermore,  from  equation  (4.6), 

ir'  -  e.'G  -  (g.,»g,o. •  •  •  ,g._)  so  that  the  expression  for  D  follows. 

-3  jl  JZ  3® 


Since  j*  e  we  use  the  general  expression  for  M  as  given  by  equation 
(5.1).  First  observe,  however,  that  G  ^e^  *  (I  -  P  +  t^e^’)^  *  (1  -  P)e^ 
+  e  -  (I  -  P)ej  *  e  so  that  Ge  ■  e^  and  hence  that  GII  ■  Getr'  ■  e ^ ir^  * . 
Consequently, 


and  equation  (5.10)  follows  from  equation  (5.1). 

(i  -  Pn)'1  9 

(c)  If,  in  equation  (5.1),  we  take  G  ■ 

0*  0 

and  ir *  »  j  (b',1),  as  given  by  equation  (4.7)  and  we  carry  out  the  requisite 

substitution,  expression  (5.11)  is  obtained. 

The  procedure  suggested  by  equation  (5.9)  has  the  advantage  of  simplicity 
but  it  does  involve  the  inversion  of  an  m  x  m  matrix  whereas  the  procedures 
suggested  by  equations  (5.10)  and  (5.11)  require  the  computation  of  an  (m  -  1) 
x  (m  -  1)  matrix  inverse.  The  expression  given  by  equation  (5.10)  has  appeared 
in  literature  for  the  case  j  ■  m  (Meyer  [12]). 


5.2.  Semi-Markov  processes 

In  section  2.2  we  saw  that  M,  the  matrix  of  mean  first  passage  times  in 
a  semi-Markov  process  satisfies  equation  (2.10),  i.e., 

(I  -  P)M  -  P(1)E  -  PD, 
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where  P 


^  *  [p±j 1  and  D  -  Md. 

This  is  a  generalization  of  the  relevant  Markov  chain  result  and  can  be 
solved  in  an  analogous  manner. 

Theorem  5.2.:  If  G  is  any  g-inverse  of  I  -  P  then 


(5.12) 


M 


tr^GP(1)n  -  -r*  e(gp(1)h)  .  +  i  -  g  +  eg,]d, 

\  \  d  d 


where  X^ 


j'y,  y 


eir' ,  D 


W 


Proof :  First  note  that  premultiplication  of  equation  (2.10)  by  tt *  yields 
nip(l)E  m  ffl|j  further  it’P^E  =  n'y  e'  *  X,e'. 

Now  if  we  follow  through  the  proof  of  Theorem  5.1  the  consistency 
conditions  are  seen  to  be  satisfied: 

[I  -  (I  -  P)G](P(L)E  -  PD)  -  [—■  -  (I  -  P)?][7r’P(1)E  -  It'D]  -  0. 

The  general  solution  is  then  given  by 


(5.13) 


M  »  G(P(1)E  -  PD)  +  [I  -  G(I  -  P)]U, 
-  G(P(1)E  -  PD)  +  eb' f 


using  equation  (5.3).  Replacing  eb’  by  EB,  as  in  the  proof  of  Theorem  5.1, 
and  taking  diagonal  elements  of  equation  (5.13)  yields 

(5.14)  D  -  (GP(1)E)d  -  (GP)dD  +  B. 

Since  GP^E  ■  -r-  GP^eir'D  ■  -r-  GP^HD  substitution  of  the  expression 
1  ”  A1 

for  B  obtained  from  equation  (5.14)  into  equation  (5.13)  yields 


expression.  □ 

An  Immediate  observation  is  that  if  *  P  and  hence  y  ■  e  and  *  1, 

as  is  the  case  for  a  discrete  time  Markov  chain,  equation  (5.12)  becomes 
equation  (5.1). 

Theorem  5.2  presents  a  new  result.  A  special  case  when  G*Z*[I-P+n]‘ 
appears  in  Hunter  [8] . 

As  was  the  case  in  section  5.1  it  is  of  interest  to  know  whether  any 
simplification  of  expression  (5.12)  occurs  when  G  has  some  additional  structure. 
Corollary  5.2.1.:  If  G  *  [I  -  P  +  yu'  ]  *  +  ef '  (with  u'e  +  0)  then 

(5.15)  M  -  [I  -  G  +  EGdJD. 

Proof : 

GP(1)JI  -  Gutt'  -  [I  -  P  +  yu'  r1^’  +  ef'iiTT'  , 

-  +  f'ylll,  using  equation  (3.17), 

■  en,  say. 

Thence  E(GP^H)d  •  GP^H,  as  in  the  proof  of  Corollary  5.1.2.  O 

An  interesting  observation  is  that  this  same  form  of  g-lnverse  for  I  -  P, 
with  f'  ■  O',  appeared  in  Theorem  4.2  when  the  limiting  distribution  of  the 
semi-Markov  process  was  considered.  Further  simplicatlon  occurs  for  D  in  this 
case  and  all  properties  of  interest  appear  in  a  simple  form. 


1 

1 
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T 


Corollary  5.2.2.:  Let  G  ■  {1  -  P  +  yu*  ]  ^  (with  u’e  j*  0),  then 
Ax  -  (u'Ge)-1,  j'  -  A^’G,  y'  -  u'G(§u,>d  , 

(5.16)  M  -  [I  -  G  +  EGdJ[eu'G)d]~1. 

Proof:  Since  n'  *  u'G/u'Ge,  (equation  (4.21))  and  u'Gy  *  1,  (equation  (3.17)) 

the  results  follow  easily.  □ 

The  techniques  used  in  this  section  can  be  extended  to  obtain  expressions 

for  M  ■  ]  for  r  _>  1.  This  problem  was  examined  by  Hunter  [8]  and 

(2\  ( 9 ) 

explicit  expressions  were  obtained  for  Md  and  M  in  terms  of  Z,  the 

fundamental  matrix.  Actually  the  methods  used  in  this  aforementioned  paper 

(2) 

extend  naturally  to  give  the  following  result  for  Md  ' .  The  proof  follows 
as  for  Theorem  2.17,  Hunter  [8]  but  with  Z  replaced  by  G. 

Theorem  5.3. :  If  G  is  any  g-inverse  of  I  -  P  then 

(5.17)  M<2>  -  y-[2{^-(IIP(1)GP(1)n)d  -  (GP(1)n)d  -  (nP(1)G)d  +  AjG^D 

+  a2i]d, 

where  A^  -  j'P^e  -  n'  y  and  A2  "  ir'P^e  -  ir'y^2\  □ 

Simplification  of  expression  (5.17)  can  be  effected  by  taking  special 
forms  for  G.  Firstly  we  use  the  form  considered  in  Corollary  5.2.1. 

Corollary  5.3.1.:  If  G  *  (I  -  P  +  yu' ]  *  +  ef '  (with  u'e  t  0)  then 

(5.18)  M^2)  -  -^•t2{A1Gd  -  (HP(1)G)d)D  +  A2I]D. 

Proof :  From  the  proof  of  Corollary  5.2.1,  GP^n  -  811  so  that 
HP(1)GP(1)n  “  Se^'P^eir'  “  BA^H.  Consequently  the  first  two  terms  of 


i 
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expression  (5.17)  csncel.  □ 

Further  simplification  can  be  effected  by  taking  a  specific  choice  of  u. 
Corollary  5.3.2.:  If  G  -  [I  -  P  +  yii'P^]-1  then 

(2)  ^2  ~  ^ 

(5.19)  Mj  ;  -  [2GdD  + - - -  I]D. 

Proof:  With  G  so  chosen  observe  that  from  equation  (3.13) 


ej'P(1)[I  -  P  +  yir'P^r1  -  j-  n  , 


and  equation  (5.19)  follows  equation  (5.18)  upon  noting  IT _D  ■1,1.  O 

a  J. 

The  g-inverse  of  1  -  P  used  in  Corollary  5.3.2  has  not  appeared  in  the 
literature  before.  It  appears  to  have  the  Interpretation  as  the  "fundamental 
matrix  of  Markov  renewal  process"  since  it  is  a  generalization  of  Kemeny  and 
Snell's  Z  matrix.  When  the  process  degenerates  to  a  Markov  chain  *  P, 

U  ■  e  and  thus  yir'P^  ■>  ex'  and  G  becomes  Z.  Also  in  this  case 

**»  ^ 

(2)  ( 2 ) 

P  ■  P,  y  -  e,  =  >2  ■  1  and  equation  (5.19)  becomes 

( 2 ) 

Md  -  [2ZdD  -  I]D  , 


as  obtained  for  Markov  chains  by  Kemeny  and  Snell  [9]. 


5.3.  Markov  chains  in  continuous  time 

In  this  case  y  -  -Q.^e  and  the  g-inverse  of  I  -  P  used  in  Corollary 
5.2.2  becomes  tl  -  P  -  Qd*eu' ]  If  we  replace  u'  by  -u'  we  can  take  a 
g-inverse  of  I  -  P  as 

G  -  [I  -  P  +  q“1«u'  f 1  -  [Q  +  eu'  f  1Qd  , 
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using  equation  (2.15).  However  [Q  +  eu* )  ^  is  a  g- inverse  of  the 
infinitesimal  generator  Q  which  we  used  in  Theorem  4.3  to  obtain  simple 
results  for  v'.  In  this  case  we  can  express  M  in  terms  of  properties  of 
Q. 

Theorem  5.4. :  If  H  *  [Q  +  eu' ]  *  (with  u'e  j*  0)  then 
(5.20)  M  -  [H  -  EHd  -  Q^Keu'Hjj]"1, 

and  v'  =•  u'H,  *  -(y'Qde)  1,  y'  ■ 


Proof :  From  Corollary  5.5.1,  since  G  ■  HQd 


M  -  [I  -  HQd  +  EHdQd]D, 


(5.21) 


-1 


lQd  -  H  +  EHdl  V* 


Now,  from  equation  (4.13),  v'  =  u'H.  This  then  implies  that,  since 

» -  -tf. 


v'A'1 


v'Q, 


it'  - 


v'  A_1e  v'Qde 


Also,  since  y  *  Ae, 


\  -  y'y 


v'A-1!!  v'e 


-y'Qde  -v'Qde  -v'Qde 


Consequently,  y'  *  -X^v'Qd.  Furthermore 


nd  “  -Ai<«y'>dQd  "  "Xi(?y’H)dQd ’ 
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6.1.  Markov  chains  in  discrete  time 

In  section  2.1  we  saw  that  the  behavior  of  the  expectations  of  the 
random  variables  and  ^  was  intimately  connected  with  sums  of  power 

of  the  transition  matrix.  The  following  theorem  shows  that  we  can  use 
g- inverses  to  obtain  expressions  for  such  sums. 

Theorem  6.1. :  If  G  is  any  g-inverse  of  I  -  P  where  P  is  an  irreducible 
transition  matrix,  then 

nil  +  (I  -  n)G(I  -  Pn), 
nil  +  (I  -  Pn)G(I  -  D). 

n-1 

Proof:  Let  A  •  E  Pr  then 
n  r-0 

(6.2)  (I  -  P)An  -  I  -  Pn, 
and 

(6.3)  A  (1  -  P)  -  I  -  Pn. 

n 

Equations  (6.2)  and  (6.3)  are  in  a  suitable  form  for  applying  Corollaries 


(6.1) 


n-1 

E 

r-0 


E  Pr  - 


3.1.1  and  3.1.2,  respectively.  Both  equations  are  in  fact  consistent  and  the 
arbitrary  constant  matrix  in  each  solution  can  be  eliminated  using  the 


observation  that  the  are  constrained  by  the  relationships 

(6.4)  A  n  -  IIA  -  nn. 

n  n 

With  G  -  [I  -  P  +  tu']  ^  +  ef*  +  gir'  it  is  easily  seen,  as  in  the  proof 
of  Theorem  5.1,  that 


t 

(6.5)  I  -  (I  -  P)G  ■  (—  -  (I  -  P)g]*’  =  on'  , 

IT 1  t 


and 

u' 

(6.6)  I  -  G(I  -  P)  =  e[—  -  f'(I  -  P)]  -  eg'. 

u’e 

We  shall  consider  only  equation  (6.2).  The  procedure  follows  analogously 
for  equation  (6.3).  The  consistency  condition  is  verified  by  using 
equation  (6.5)  while  equation  (6.6)  shows  that  the  general  solution  is 
given  by 


(6.7) 


A  -  G(I  -  P°)  +  eg'U  , 
n  —  n 

-  G(I  -  Pn)  +  eun’ ,  say. 


Further,  from  equation  (6.5) 

(6.8)  nil  -  JIG(I  -  Pn)  +  eun'»  (since  He  ■  e). 


Elimination  of  eu  '  between  equations  (6.7)  and  (6.8)  leads  to  the  first 
— n 

fora  of  equation  (6.1)  for  Aq. 

If  we  restrict  attention  to  regular  Markov  chains  then  lim  p^  -  ir 

n-+«  J  ' 

for  all  l,j  -  1,2, '“.m  and  thus,  in  terms  of  matrices  11m  Pn  -  n.  With 


this  observation  we  obtain  the  following  information  concerning  the  behavior 
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of  £  F  for  large  n. 
r-0 

Corollary  6.1.1.:  If  G  is  any  g-inverse  of  I  -  P  where  P  is  the  transition 
matrix  of  a  regular  Harkov  chain  then 
n-1  r 

(6.9)  £  P  -  nil  +  (I  -  n)G(I  -  n)  +  o(l)E. 
r-0 

Corollary  6.1.2.:  If  G  -  (I  -  P  +  tu')  ^  +  ef'  +  gir'  (t  t  0  and  u'e  ^  0) 
then  if  P  is  the  transition  matrix  of  a  regular  Markov  chain 

(6.10)  -  nit  +  (I  -  n)(I  -  P  +  tu')_1(I  -  It)  +  o(l)E. 
r-0 

Proof:  Equation  (6.10)  follows  upon  the  required  substitution  and  observing 

that  (I  -  n)e  -  0  and  ir'  (I  -  It)  -  O'. 

In  terms  of  the  expectations  of  the  random  variables  and  ,  the 

ij  ij 

above  corollaries,  in  conjunction  with  equation  (2.7),  give  immediately: 

Theorem  6.2.:  For  regular  Markov  chains 

(6.11)  [EM^]  -  (n  +  1)11  +  (I  -  n)G(I  -  n)  +  o(l)E, 
and 

(6.12)  [EN*^]  -  (n  +  l)n  -  I  +  (I  -  n)G(I  -  II)  +  o(l)E. 

Although  the  general  expression  of  Theorem  6.2  are  new,  if  G  is  replaced 
by  Z,  the  fundamental  matrix,  we  obtain  the  results  of  Kemeny  and  Snell  [9] 
while  if  G  is  replaced  by  A  ,  the  group  inverse  of  I  -  P,  we  obtain  the  results 
of  Meyer  [11).  In  these  special  cases  we  get  simpler  forms.  In  particular  we 
have  the  following  corollary. 


Corollary  6.2.1.:  For  a  regular  Markov  chain 

nil  +  Z  +  o(l)E, 

(6.13)  [EM^]  - 

(n  +  l)n  +  A#  +  o(l)E. 

i 

Proof :  Simplification  of  equation  (6.11)  gives  the  required  forms  after 
observing  that  (I  -  H)Z(1  -  II)  «=  Z  -  H  and  (I  -  H)A#(I  -  H)  -  A#.  □ 

6.2.  Markov  renewal  processes 

In  section  2.2  an  asymptotic  form  for  the  Markov  renewal  kernel,  M(t) 
was  given  by  equation  (2.12)  i.e., 

M(t)  -  tE(Md)_1  +  ~  E[(Md)'1]2M^2)  -  M(Md)_1  +  o(l)E. 

By  utilizing  the  results  of  Theorems  5.2  and  5.3  and  their  corollaries 
we  are  able  to  obtain  expression  for  M(t)  in  terms  of  g-inverses  of  I  -  P, 
where  P  is  the  transition  matrix  of  the  imbedded  irreducible  Markov  chain. 

Theorem  6.3, :  If  G  is  any  g-inverse  of  I  -  P  then 

(6.14)  M(t)  «-pn+-^-n  +  [i -  7=-  nP(1)]G[i  -  ~  p(1)n]  -  i  +  o(i)E, 

A1  X^  A1  1 

where  X^  ■  r’P^e,  Xj  ■  rr'P^e  . 

Proof:  As  for  the  proof  of  Theorem  3.3  of  Hunter  [ 8  ]  but  with  Z  replaced 
by  G.  O 

Equation  (6.14)  is  identical  in  form  to  that  obtained  earlier  with  the 
special  g-inverse,  Z,  the  fundamental  matrix  of  the  imbedded  Markov  chain. 

However  the  generality  of  Theorem  6.13  enables  us  to  use  special  forms  of  G 
to  obtain  simpler  structural  results. 
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-1 


loroi-Lary  o.j.i.:  (a)  ir  u  =  [i  -  r  +  yu’ j  ^witn  u  e  f  u;  tnen 


(6.15) 


M(t)  =  t5-  n  +  n  +  [i  -  np(1)]G  -  i  +  o<i)e. 

i  a  2  Ai 


(b)  If  G  =  (I  -  p  +  tTT'P^]-1  (With  t  *  0)  then 


(6.16) 


M(t)  =  TLII+-^r  +  G[I--i-  P(1)n]  -  I  +  o(l)E. 
A1  A^  A1 


(c)  If  G  -  [1  -  P  +  yn'P(1)]-1  then 

(A,-i)n 

(6.17)  M(t)  n+  ,  ■  +  G  -  I  +  o(l)E. 

A1  A^ 

Proof :  If  G  *  [I  -  P  +  tu" ]-1  then 

if  t  ■  y  ■  P^^e,  equation  (3.17)  implies  GP^e  =  ^7^-  , 

(1)  (1) 
while  if  u'  =  ir'P  ,  equation  (3.13)  implies  tt'Pv  'g  =  ^  . 

With  the  appropriate  substitution  for  t  and  y  in  equation  (6.14)  the 
stated  results  are  obtained  following  simplification.  □ 

The  expressions  given  in  Theorem  6.3  and  its  corollaries  are  new. 
Computationally  equation  (6.14)  offers  much  more  flexibility  than  previously 
thought  possible  whereas  equation  (6.17)  offers  a  structurally  simple  result. 

Observe  that  when  the  process  degenerates  to  a  Markov  chain  A^  *  Aj  *  1, 
yrr,Pl'i'  *  err’  and  equation  (6.17)  reduces  Immediately  to  the  form 
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(6.18) 


M(n)  -  [EN^  ]  =  nil  +  Z  -  I  +  o(l)E  , 

which  follows,  alternatively,  from  equations  (2.7)  and  (6.13). 

We  have  not  examined  expressions  for  the  higher  moments  of  Markov 
renewal  counting  processes.  A  discussion  concerning  the  computation  of 
variances  is  given  in  Hunter  [  8  ] . 

6.3.  Markov  chains  in  continuous  time 

In  this  case  the  results  of  Theorem  6.3  and  its  corollary  apply  with 
P(1)  -  -Q"1?  and  P(2)  *  2(Q~Vp  implying  that  ^  =  -w'Q^e  =  -(v'C^e)-1 
and  X2  -  2tt'  (Q~L)2e  =  -2X1v'Q^1e. 

Consequently  we  can  derive  expressions  for  M(t)  involving  the 
stationary  probability  vector  v'  and  generalized  inverses  of  Q  for 
irreducible  Markov  chains  in  continuous  time. 

Theorem  6.4.:  If  H  is  any  g-inverse  of  Q  then 

(6.19)  M(t)  =  -tLQd  +  2tLQd  +  [I  -  LP]H[I  -  L]Qd  -  I  +  o(l)E, 

where  L  =  ev'  and  x  =  v'Q.^e 
*■’  ~  -  a  - 

Proof :  If  H  is  any  g-inverse  of  Q  then,  using  equation  (3.20)  and  the 
observations  that  I  -  P  ■  Q.^Q  and  it’  =  -X.v'Q,,  it  is  easily  seen  that 
H  -  GQ^  wiiere  G  is  a  g-inverse  of  I  -  P. 

With  L  ■  ev'  it  is  easily  seen  that  II  ■  -X^LQd>  PL  *  L  and  thus 
equation  (6.19)  follows  from  equation  (6.14). 

Analogous  results  for  the  special  cases  of  Corollary  6.3.1  can  be 
examined  by  taking  H  as  (Q  +  tv'P)-1,  (Q  +  eu')-1  and  (Q  +  LP)"1. 


7.  Concluding  remarks 

In  a  paper  examining  the  role  of  group  generalized  inverses  in  the 
theory  of  finite  Markov  chains,  Meyer  [11]  stated  that  the  "correct" 
generalized  inverse  to  use  is  the  group  generalized  inverse.  He  also 
claimed  that  the  Moore-Penrose  inverse  or  any  other  g-inverse  that  satisfies 
a  subset  of  the  Penrose  conditions  of  Definition  3.2  are  the  wrong  types  since 
they  are  "forced"  into  the  theory  because  of  their  ability  to  produce 
solutions  of  consistent  systems  of  linear  equations.  However,  as  we  have 
exposed  in  this  paper,  by  taking  the  class  of  all  g-inverses  as  our  starting 
point  we  are  able  to  present  a  unified  approach  to  a  large  class  of  different 
but  related  problems  for  a  wide  variety  of  Markovian  and  semi-Markovian  models. 

If  any  one  g-inverse  stands  out  it  is  the  matrix  [I  -  P  +  pir'P^]  *  as 
presented  for  the  first  time  in  Corollary  5.3.2  and  later  in  Corollary  6.3.1(c). 
As  we  saw,  this  matrix  is  a  generalization  for  semi-Markovian  processes  of  the 
fundamental  matrix  of  irreducible  Markov  chains  as  first  presented  by  Kemeny  and 
Snell  [9], 

Of  course,  the  group  Inverse  has  other  advantages  when  one  is  Interested  in 
classifying  states  of  the  Markov  chains  but  Campbell  and  Meyer's  claim  [3]  that 
some  types  of  g-inverses  lead  to  cumbersome  expressions  which  do  little  to 
enhance  or  unify  the  theory  and  provide  no  practical  or  computational  advantage 
is  disputed.  The  observation  that  all  known  explicit  methods  for  finding 
stationary  distributions  can  be  put  in  a  generalized  inverse  framework  together 
with  the  result  that  any  g-inverse  can  be  used  in  examining  the  problems 
presented  in  this  paper  is  a  compelling  reason  that  g-inverses  have  an  important 
role  to  play  with  a  wide  variety  of  computer  subroutines  available  for  g-inverses. 


many  included  in  statistical  package 
large  arsenal  of  techniques  availabl 
computational  aspects  of  this  study. 
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